Real-time marine snow noise removal from underwater video

ABSTRACT

Optical flow refers to the pattern of apparent motion of objects, surfaces, and edges in a visual scene caused by the relative motion between an observer and a scene. Optical flow algorithms can be used to detect and delineate independently moving objects, even in the presence of camera motion. The present invention uses optical-flow algorithms to detect and remove marine snow particles from live video. Portions of an image scene which are identified as marine snow are reconstructed in a manner intended to reveal underwater scenery which had been occluded by the marine snow. Pixel locations within the regions of marine snow are replaced with new pixel values that are determined based on either historical data for each pixel or a mathematical operation, such as one which uses data from neighboring pixels.

PRIORITY CLAIMS

This application claims the benefit of U.S. Provisional Application Ser.No. 62/981,157 filed on Feb. 25, 2020, the contents of which areincorporated herein by reference.

FIELD OF THE INVENTION

The present invention relates to real-time video enhancement

BACKGROUND

Underwater video is of broad interest in areas such as equipmentinspections, surveillance, search & rescue, mine countermeasures, anddeep-sea exploration. However, the acquisition of underwater imagespresents a new set of challenges compared to air space images. Ingeneral, underwater visibility tends to be poor due to conditionsincluding poor natural light at different depths, the presence ofsuspended particles and the consequent scattering of light in alldirections. Thus, underwater imagery suffers from poor quality and lossof conveyed information resulting from contrast and color decay, lightscattering, blur, haze, and various types of noise. There are numerousimage processing methods that can filter out these unwanted effects.However, marine snow is a special type of noise which can profoundlydegrade the quality of underwater images and is difficult to filter out.

Marine snow is a phenomenon caused by light back scattering from smallorganic and mineral particles and air bubbles. These particles tend togrow as they fall down through the water and show up in images as brightspots of various shapes and sizes, which resemble snowflakes.

The problem of filtering out marine snow has been difficult to addressbecause the particles can be quite large and have different structuraland lighting characteristics that make them fundamentally different fromother types of noise encountered in digital images. Thus, there is aneed for a method for marine snow filtering, such as the one presentedherein.

SUMMARY OF INVENTION

The present invention seeks to provide a solution to this problem byusing optical flow algorithms to remove visual occlusions caused bymarine snow in live video.

These and other aspects, objects, features and advantages of the presentinvention, are specifically set forth in, or will become apparent from,the following detailed description of an exemplary embodiment of theinvention.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flow diagram of the video processing platform architecture.

FIG. 2 is an image of a live underwater video stream.

FIG. 3 is an image of a Synthetic Video Frame.

FIG. 4 is a diagram of the Dynamic Chroma Mask.

FIG. 5 is an image of an Augmented Live Video Frame.

FIG. 6 is a flow diagram of the Dynamic Chroma Mask.

FIG. 7 is a flow diagram of the Synthetic Video Frame.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

In a preferred embodiment, the present invention uses chroma key maskscreated from optical flow algorithms to remove visual occlusions causedby marine snow in live video. Chroma key is a technique used in video toreplace a portion of an image with a new image. For example, chroma keyis commonly used in the film industry to replace a monochromebackground, such as a green screen, with a different setting. Thepreferred embodiment uses a chroma key operation to merge pixels fromthe live video stream with pixels from a synthetic video frame todeliver a video frame with no occluded pixels. The live video framecontains clusters of “snow pixels” that are extracted using optical flowalgorithms to create a chroma mask for use in extracting unoccluded livepixels. In essence, this mask replaces particles of marine snow withineach video frame with similarly shaped regions of a single color.

The synthetic video contains live unoccluded pixels extracted from thecurrent video frame and unoccluded pixels carried forward from previousframe(s). The merging operation is controlled by a dynamic chroma maskcreated for each frame by optically identifying and tracking the “snow”particles. The chroma mask chooses a live pixel if not hidden behind a“snow” pixel, otherwise it selects the pixel from the synthetic videoframe.

The present invention provides a method for removing certain visualocclusions referred to as marine snow from live underwater video usingoptical flow algorithms. The preferred embodiment uses optical flowalgorithms to create chroma key masks that are used to remove marinesnow and merge pixels from the live video stream with pixels from asynthetic video frame to deliver a video with no occluded pixels. Themethod is described in detail in diagrams and associated text thatfollows.

For the purposes of this description, image pixels can be defined toexist in one of the following four states: 1) a Live pixel is visible,unoccluded; 2) a Snow pixel is an unwanted waste particle that floats inthe frame; 3) a Hidden pixel is one that is hidden by Snow; 4) a Historypixel is one that was previously visible but now occluded in the currentframe.

There are five basic steps comprising the method for removing marinesnow:

-   1) The live video frame contains clusters of “snow pixels” that are    extracted using optical flow algorithms to create a chroma mask for    use in extracting unoccluded live pixels.-   2) The synthetic video contains live unoccluded pixels extracted    from the current video frame and unoccluded pixels carried forward    from previous frame(s).-   3) The merging operation is controlled by a dynamic chroma mask    created for each frame by optically identifying and tracking the    “snow” particles.-   4) The chroma mask chooses a live pixel if not hidden behind a    “snow” pixel, otherwise it selects the pixel from the synthetic    video frame.-   5) A video frame with no occluded pixels is created as the chroma    key operation merges pixels from the live video stream with pixels    from a synthetic video frame.

FIG. 1 shows an example architecture of a video processing platform 100that could be used to implement the method for removing marine snow fromvideo streams. It shows a video input 114 for receiving a video inputstream from a connected video camera 110 or video archive 108. Alsoshown are a connected video display, a DVR storage module and a networkmodule to which the processed video is output 116. The video processingplatform 100 integrates a CPU 102 for command and control, a GPU 104 forimage processing through a user interface 112, and an FPGA 106 to runthe optical flow algorithms 118 in real time. A video controller is alsoincluded (not shown) which provides the capability to fuse incomingvideo with the chroma masks and to encode/decode the IP video streams.

FIG. 2 shows a live underwater video stream that contains snow pixelswhich hide live pixels.

FIG. 3 shows a Synthetic Video Frame which consists of unoccluded LivePixels from the current video frame and History Pixels carried forwardfrom previous frames. There are no Snow Pixels because they have beenreplaced with History Pixels carried forward from previous frames.

FIG. 4 shows a Dynamic Chroma Mask 400 created for each frame usingOptical Flow algorithms 402 to identify and track floating pixelclusters (“Snow Pixels”) 404. Snow Pixel clusters are geo-positioned onthe mask in the chroma key 406 color.

FIG. 5 shows an Augmented Live Video Frame wherein Snow Pixels have beenreplaced by History Pixels selected from the Synthetic Video Frame.

FIG. 6 shows how the Dynamic Chroma Mask 600 is created for each frameto filter out the Snow Pixels from the live video. A Dynamic Chroma Mask600 is created for each frame using Optical Flow algorithms 602 toidentify and track floating pixel clusters (“Snow Pixels”) 604. SnowPixel clusters are geo-positioned on the mask in the chroma key color.The chroma mask will be used to filter out clusters of Snow Pixels fromthe live video 606.

FIG. 7 shows how the Synthetic video frame 700 is updated as the lastoperation to prepare for the next frame. When updating the Syntheticvideo frame through a Dynamic Chroma Mask 702 using inverted chromakeying 704, only unoccluded pixels 706 are extracted from the live videoframe 708 to update the synthetic frame 700. The pixels in the syntheticframe that are currently occluded by the snow pixel clusters in the liveframe remain and carry forward.

In certain embodiments, pixels in the synthetic frame that are not fromthe current frame may comprise pixels convolved from neighboring pixelsinstead of carrying forward the last instance where the pixel wasvisible.

While various embodiments of the disclosed technology have beendescribed above, it should be understood that they have been presentedby way of example only, and not of limitation. Likewise, the variousdiagrams may depict an example architectural or other configuration forthe disclosed technology, which is done to aid in understanding thefeatures and functionality that may be included in the disclosedtechnology. The disclosed technology is not restricted to theillustrated example architectures or configurations, but the desiredfeatures may be implemented using a variety of alternative architecturesand configurations. Indeed, it will be apparent to one of skill in theart how alternative functional, logical or physical partitioning andconfigurations may be implemented to implement the desired features ofthe technology disclosed herein. Also, a multitude of differentconstituent module names other than those depicted herein may be appliedto the various partitions. Additionally, with regard to flow diagrams,operational descriptions and method claims, the order in which the stepsare presented herein shall not mandate that various embodiments beimplemented to perform the recited functionality in the same orderunless the context dictates otherwise.

Although the disclosed technology is described above in terms of variousexemplary embodiments and implementations, it should be understood thatthe various features, aspects and functionality described in one or moreof the individual embodiments are not limited in their applicability tothe particular embodiment with which they are described, but instead maybe applied, alone or in various combinations, to one or more of theother embodiments of the disclosed technology, whether or not suchembodiments are described and whether or not such features are presentedas being a part of a described embodiment. Thus, the breadth and scopeof the technology disclosed herein should not be limited by any of theabove-described exemplary embodiments.

Terms and phrases used in this document, and variations thereof, unlessotherwise expressly stated, should be construed as open ended as opposedto limiting. As examples of the foregoing: the term “including” shouldbe read as meaning “including, without limitation” or the like; the term“example” is used to provide exemplary instances of the item indiscussion, not an exhaustive or limiting list thereof; the terms “a” or“an” should be read as meaning “at least one,” “one or more” or thelike; and adjectives such as “conventional,” “traditional,” “normal,”“standard,” “known” and terms of similar meaning should not be construedas limiting the item described to a given time period or to an itemavailable as of a given time, but instead should be read to encompassconventional, traditional, normal, or standard technologies that may beavailable or known now or at any time in the future. Likewise, wherethis document refers to technologies that would be apparent or known toone of ordinary skill in the art, such technologies encompass thoseapparent or known to the skilled artisan now or at any time in thefuture.

What is claimed is:
 1. An optical flow sensor to detect the fluctuationsin water while immersed within bodies of said water comprising: a camerafor detecting optical flow events within said water; an optical flowcomparison process wherein independently moving objects in proximity ofsaid body of water are isolated against a backdrop of water flow notassociated with said moving objects and wherein movement of said camerais subtracted from movement associated with said object; a processperformed by a computer wherein video signals produced by said camerawherein marine snow noise is identified and removed from an overallunderwater image, so that said overall underwater image consists of saidmoving objects and wherein said overall underwater image is relativelyclear for human observation, and wherein underwater scenery otherwiseoccluded by marine snow noise become more clearly visible for humanobservation.
 2. An optical flow sensor according to claim 1 whereinpixel locations within water columns with distributions of marine snownoise are replaced with new pixel values that are determined based oneither historical data for each pixel or a mathematical operation, suchas one which uses data from neighboring pixels by way ofgeo-positioning.
 3. A method for an optical flow sensor to detect thefluctuations in water while immersed within bodies of said water andproducing a corresponding data output comprising: a camera for detectingoptical flow events within said water; an optical flow comparisonprocess wherein independently moving objects in proximity of said bodyof water are isolated against a backdrop of water flow not associatedwith said moving objects and wherein movement of said camera issubtracted from movement associated with said object; a processperformed by a computer wherein video signals produced by said camerawherein marine snow noise is identified and removed from an overallunderwater image, so that said overall underwater image consists of saidmoving objects and wherein said overall underwater image is relativelyclear for human observation, and wherein underwater scenery otherwiseoccluded by marine snow noise become more clearly visible for humanobservation.
 4. A method according to claim 3 wherein pixel locationswithin water columns with distributions of marine snow noise arereplaced with new pixel values that are determined based on eitherhistorical data for each pixel or a mathematical operation, such as onewhich uses data from neighboring pixels by way of geo-positioning.
 5. Asystem for providing clear underwater visualization using an opticalflow sensor to detect the fluctuations in water while immersed withinbodies of said water comprising: a camera for detecting optical flowevents within said water; an optical flow comparison process whereinindependently moving objects in proximity of said body of water areisolated against a backdrop of water flow not associated with saidmoving objects and wherein movement of said camera is subtracted frommovement associated with said object; a process performed by a computerwherein video signals produced by said camera wherein marine snow noiseis identified and removed from an overall underwater image, so that saidoverall underwater image consists of said moving objects and whereinsaid overall underwater image is relatively clear for human observation,and wherein underwater scenery otherwise occluded by marine snow noisebecome more clearly visible for human observation, and wherein pixellocations within water columns with distributions of marine snow noiseare replaced with new pixel values that are determined based on eitherhistorical data for each pixel or a mathematical operation, such as onewhich uses data from neighboring pixels.
 6. A system according to claim5 wherein said neighboring pixels include at least said historical datafor each of said pixels and wherein said historical data is shared withother third party users of said system's user interface.